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EXECUTIVE SUMMARY 


Nn 


The Science of Learning and Readiness (SoLaR) project seeks to demonstrate to Defense and 
other Government stakeholders the “art of the possible” for high-quality distributed learning and 
to create a practical guide for how to infuse such qualities into the broader Department of 
Defense (DoD) distributed learning ecosystem. This report is the third in a series of three reports 
examining the state of the art for advancing the applications of distributed learning. This report 
provides recommendations for setting up modern learning ecosystems that incorporate 
distributed learning techniques. The report consists of specific, actionable recommendations the 
DoD can use for enhancing its distributed/blended learning institutions (i.e., the overall system), 
courseware, and associated learning strategies/tactics. 


This report provides recommendations on best practices for supporting and creating modern 
learning ecosystems. The recommendations are based on findings from our two previous reports. 
The first was a review of the state of the art for distributed learning environments that searched 
for best practices with evidence of supporting learning and improving learning organizations. 
The second report was a review of exemplar learning organizations that have successfully grown 
their learning enterprise. The summarized findings from our review of best practices are 
provided below as four high-level results: 


» The basic principles of human learning from the learning sciences still hold within 
learning-at-scale environments. 

> Human learning within these environments must be supported by technology. 

» The technology must provide data on the learning process to the learning organization’s 
stakeholders. 

>» Learning organizations must use data to support: 1) the learners in the form of learning, 
social, and academic guidance, and 2) the members of the learning institutions for 
providing training, support, and recognition. 


Our findings from interviews of exemplar learning organizations are summarized below: 


> A “think-before-you-act” mentality is needed for successfully growing the learning 
initiative. Detailed planning and flexible policies should be in place before and as the 
organization grows. 

>» A technological infrastructure is needed that provides data on learner success and staff 
training and performance. 

» Training is needed to keep all stakeholders on the same page. 

> Knowledgeable, responsive leadership and cross-sector strategies/policies are needed to 
successfully grow. 


Description of Report 


The findings of the two previous reports have been condensed into a series of recommendations 
at the institution and course creation level. At the institution level, we provide eight specific 
recommendations. These recommendations include human-level strategies (e.g., educational 
networks, human-centered evaluations, facilitating trust) and organization support-level 
strategies (e.g., support services, humans and technological support infrastructures, and class size 
flexibility). At the course level, we provide eight recommendations that deal with course 
infrastructure (e.g., data support, facilitating self-regulation, supporting motivation, facilitating 
interaction and facilitating engagement/creativity with Community of Inquiry ideas) and content 
design (e.g., scaffolding, cognitive constraints, video integration). The recommendations are 
supported by two heuristic checklists that experts (independent external experts or internal 
members of the organization educational network) can use to evaluate the recommendations at 
the organization and course levels. 


RECOMMENDATION REPORT: 
SCIENCE OF LEARNING PRACTICES FOR 
DISTRIBUTED ONLINE ENVIRONMENTS 


This report provides recommendations on the best practices for supporting distributed online 
environments. This report provides 16 recommendations based on the science of learning 
literature and lessons learned from exemplar organizations. It provides examples of these 
recommendations. Finally, we provide two heuristic checklists that can be used to make 
observations of learning organizations and courses for compliance with the recommendations. 


RECOMMENDATIONS FOR LEARNING ORGANIZATIONS 


Learning organizations are rapidly adapting how they provide education and training. These 
changes are both technology-driven and practical to provide education and training to greater 
numbers of learners at a rapid pace. Many of these learners are immersed in online learning 
environments. For example, there are estimated to be 6,651,536 students enrolled in online 
education courses at the postsecondary level. These 6.7 million students account for 33.7% of the 
current student population and are part of a growth trend of online learning which has continued 
for the last 13 years in the United States. 


ORIGINS OF THE RECOMMENDATIONS 


This report provides recommendations on best practices for supporting and creating modern 
learning ecosystems. The recommendations are based on findings from our two previous reports. 
The first was a review of the state of the art for distributed learning environments that searched 
for best practices with evidence of supporting learning and improving learning organizations. 
The second report was a review of exemplar learning organizations that have successfully grown 
their learning enterprise. The summarized findings from our review of best practices are 
provided below as four high-level results: 


1) The basic principles of human learning from the learning sciences still hold within 
learning at scale environments. 

2) Human learning within these environments must be supported by technology. 

3) The technology must provide data on the learning process to the learning organization. 

4) Learning organizations must use data to support both learners with learning, social, and 
academic guidance, and organization members by providing needed training, support, 
and recognition. 


Our findings from interviews with exemplar learning organizations are summarized below: 


1) A “think-before-you-act” mentality is needed for successfully growing the learning 
initiative. Detailed planning and flexible policies should be in place before and as the 
organization grows. 

2) A technological infrastructure is needed that provides data on learner success and staff 
training and performance. 

3) Training is needed to ensure all stakeholders share a common understanding. 

4) Knowledgeable, responsive leadership and cross-sector strategies/policies are needed to 
grow successfully. 


SUMMARY OF BEST PRACTICE RECOMMENDATIONS 


The findings of the two previous reports have been condensed into a series of recommendations 
at the institution and course creation level. At the institution level, we provide eight specific 
recommendations. These recommendations include human-level strategies (e.g., educational 
networks, human-centered evaluations, facilitating trust) and organization support-level 
strategies (e.g., support services, humans and technological support infrastructures, and class size 
flexibility). At the course level, we provide eight recommendations that deal with course 
infrastructure (e.g., data support, facilitating self-regulation, supporting motivation, facilitating 
interaction and facilitating engagement/creativity with Community of Inquiry ideas) and content 
design (e.g., scaffolding, cognitive constraints, video integration). The recommendations are 
supported by two heuristic checklists that experts (independent external experts or internal 
members of the organization educational network) can use to evaluate the recommendations at 
the organization and course levels. 


INSTITUTIONAL RECOMMENDATIONS 


The following recommendations are a set of considerations needed for implemented a learning 
organization that can grow (scale up) and still meet the needs of its stakeholders and constituents. 
These recommendations could be implemented via company policy, an overall commitment to 
learning, and by building appropriate support systems into the organizations structure. 


NETWORK OF EDUCATION EXPERTISE 


A basic understanding and commitment to human learning at all levels of the organization, from 
administrators to SMEs/instructors, is crucial. This culture must be set from top-level 
administrators and supported throughout the organization. It is useful for all decision makers to 
have some level of understanding of good practices. Individuals should have detailed knowledge 
of learning principles with the trust/authority to support implementation throughout the 
institution’s network. This is not a single person; rather, it is a group of people spread throughout 
the levels of the organization, including educational specialists serving as higher level directors, 
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learning engineers, instructional designers, and SME/instructors that are domain-based 
educational researchers. 


EXAMPLE 


Examples of designing strategies for learning within an organization include: 


e Establishing a distinct set of guidelines that outline how the learning ecosystem will be 
structured and how the ecosystem will function between stakeholders. 

e Identifying the leaders and contributors to the learning ecosystem and its maintenance. 

e Allowing for learning meetings among the learning network. 

e Promoting understanding of learning with focused learning meetings or specialized 
training for all levels of management. 

e Building a learning community or learning network within the organization. Identify their 
expertise within the organization. 

e Calculating the financial impact to the organization. 

e Collecting and interpreting learning analytics to improve the learning ecosystem’s design 
and functionality. 


ESTABLISH HUMAN-CENTERED EVALUATION PROCESSES 


Organizations must understand the strengths, weaknesses and needs of all stakeholders. Modern 
learning ecosystems are large, complicated structures with diverse stakeholders. To serve the 
entire organization and make informed decisions, it is essential to understand the needs of 
distinct groups. Human-centered evaluations should be used to evaluate the functionality of 
computer and technological systems and to collect data on how humans function within a 
learning ecosystem. 
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Human-centered design and evaluation will allow 
for a better understanding of the overall 
performance of the learning organization. It 
should have useful data-gathering methods for all 
levels of deployment. Early evaluation (e.g., needs Undemtandion identifying 
assessments, contextual interviews) allows for Needs Problems 
understanding the needs of the users to better set 
requirements and design support into the system. 
Mid-level evaluation of systems (e.g., usability 
testing) allows for identification of problems that 
users will have with the system before it is 


Evaluating 
Performance 


implemented on a larger scale. Late evaluation 
(e.g., experimental testing and mixed method 
interviewing) of completed systems provides 
evidence of effectiveness. This cycle can increase confidence and engagement from end users. 
This could be applied new learning supports integrated into classes or to better understand the 
current state of areas within the organization. 


EXAMPLE 


Human-centered evaluations within eLearning courses might take various forms. For instance, a 
usability evaluation might enroll students in online class shells, and then record errors and 
navigation behaviors as they locate materials and perform tasks. Such usability tests can span 
observational methods (e.g., digital observation via screen capture software) or think-aloud 
procedures (e.g., via video conferencing) wherein students talk about what they are attempting to 
accomplish. Within a larger organization, survey methods can be used to develop an 
understanding of general knowledge or perceptions about proposed implementations. 


SUPPORT ORGANIZATIONAL TRUST 


Organizations must also foster a sense of trust at all levels. In this case, “trust” broadly 
encompasses confidence in and positive appraisals of available technologies and between 
organization members. Administrators must encourage trust, foster relationships, and seek 
common ground for discussion and action between stakeholders, while also collecting and using 
data to facilitate change and support faculty in the online education endeavor. To be 
“trustworthy,” administrative decision making should be guided by evidence-based tools and 
metrics. Evaluation of stakeholder performance should be fair and transparent, and evaluations 
should incorporate feedback from stakeholders. It should be grounded in policy that includes 
recognition of stakeholders’ contributions (e.g., compensation or acknowledgement of time 
commitment). 


EXAMPLES 
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Examples of helping establish organizational trust include: 


e Implementing clear evaluation metrics. 

e Developing performance evaluations that follow metrics and are transparent. 

e Providing clear and actionable performance feedback. 

e Recognizing stakeholders’ contributions to success. 

e Providing visible acknowledgement of effort. 

e Identifying and providing support for online instructors through technical assistance and 
course design/development assistance. 

e Establishing clear workload and compensation models based on metrics and effort. 

e Establishing course quality control methods to ensure accessibility and engagement. 


SUPPORT INSTITUTION-STUDENT TRUST 


Trust is crucial at the student level. Students’ trust in instructors and class content directly 
impacts course grades. Student trust can be established by social engineering of the learning 
environment (i.e., building logical class structures that minimize negative events), frequent 
communication, maintaining a positive and consistent instructor persona, supporting peer-to-peer 
mutual interaction (e.g., collaboration), involving students in decision making and 
communication, defining clear policies, and creating clear and transparent oversight. 


EXAMPLES 


Examples of helping establish trust include: 


e Eliminating student barriers to online learning by helping students locate and use external 
resources, such as library services, tutoring services, and technology services. 

e Minimizing anxiety over computer use and access through the provision of computer 
access and training. 

e Establishing positive rapport through respectful and fair treatment during online 
interactions. 

e Providing understandable and thorough instructions for student interactions with online 
course content by providing an up-to-date syllabus, course goals, and realistic workload 
expectations. 

e Pre-emptively designing fair evaluation measures of course and student success based on 
the nuances of online instruction. 

e Explaining course grading and evaluations prior to beginning the course. 

e Monitoring course completion and drop rates in real time to redeem struggling students. 


STUDENT ACADEMIC/SOCIAL SUPPORT SERVICES 
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Learners must have adequate academic and social support structures from the institution. As 
online distributed learning technologies continue to advance and propagate, the potential for 
isolating students is a problem that must be addressed. Virtual interactions and asynchronous 
environments may result in fewer opportunities for students to interact with peers or the 
organization in meaningful ways. This can decrease the learner’s connection to the organization 
and long-term engagement in the learning process. 


Learning organizations must offer student support services and mindfully enable additional 
social structures. There are several categories of support, such as academic services (e.g., 
advising, library, financial, and admissions) and social services (e.g., student organizations, 
psychological services, placement services, and instructor support). These services interact with 
and build upon other essential factors, including students’ family framework, personal 
satisfaction, and perceived course relevance. All these elements play critical role in students’ 
decisions to persist or drop out of online courses. 


EXAMPLES 


Examples of student social support include: 


e Providing students with 
administrative contact 


persons for individualized 


counseling. * Student organizations 


© Psychological services 
* Placement services 
¢ Instructor support 


e Instituting institution-wide 
communication standards 
and evaluate 
communication flow at 


: Academic 
regular intervals. 


services 
e Making students aware of 


financial obligations and 
sources of help throughout 
the duration of the course. 


¢ Advising services 
¢ Library services 
* Financial services Classroom 
¢ Admissions services success 


e Discussing opportunities 
for inclusion in services 
offered by institutional 
services for persons with 
disabilities. 

e Making students aware of counseling services. 

e Making library services available to all enrolled students. 
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GENERAL SUPPORT INFRASTRUCTURE 


The learning organization needs a clear hierarchy of reporting, networks of support, and training 
on technology and procedures. This support infrastructure will help to ease the burden of course 
creation on instructors and allow them to focus on higher quality content. This support includes: 


e Providing educational design support from instructional designers. 

e Providing learning strategy support from learning scientists or discipline-based 
educational researchers. 

e Providing learning technology integration support on best practices for using technology 
to support learning from learning engineers. 
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GENERAL IT INFRASTRUCTURE — INCLUDING ACADEMIC COURSEWARE, 
ANALYTICS INFRASTRUCTURE 


Technological infrastructure support should be at the core of any learning organization. Without 
dedicated planning, organizations will struggle to deliver new technology= for users while also 
(a) maintaining legacy systems beyond their reasonable lifespans, (b) seeking interoperability 
between incompatible applications, and (c) doing so with dwindling resources. This requires a 
variety of supporting infrastructures included appropriate policies and processes, information and 
communication technologies, instructional support staff, technology hardware and facilities, and 
training. This comprehensive support structure must be in place for instructors, staff, and 
students with emphasis on technology support. Ideally, a modern infrastructure must transcend 
vertical and isolated systems to embrace open data formats that can integrate data from across 
the learning enterprise. 


FLEXIBLE CLASS SIZES 


Although learning at scale aims to provide worthwhile instruction to larger numbers of learners, 
this goal does not mean that class sizes can grow infinitely. Appropriate class size is a 
complicated question that must be considered by learning organizations, which should consider 
(a) the type(s) of information being taught and (b) the technologies available to support the 
learning environment. One generally recommended “rule” is that class size should be guided by 
nature of the content (see Bloom’s taxonomy). Topics that require higher-level thinking (e.g., 
synthesis and evaluation) may be best suited to smaller class, whereas topics that entail lower- 
level thinking (e.g., recall) may be taught in larger classes. 


EXAMPLES 


Examples of implementation of class size recommendations include: 


e Planning for smaller class sizes when teaching high-level 
content. 
e Utilizing larger classes when foundational principles are the 
class focus. 
e Limiting class sizes, regardless of content, when learner or 
instructor support is limited. 
e Scaling courses at a commiserate pace with student As material complexity increases class size should 
engagement ability. anes 
e Planning for transition time and course re-development of 
currently face-to-face courses. 
e Optimizing technology for maintaining student interaction. 
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COURSE CREATION RECOMMENDATIONS 


Course creation is going to be one of the most important expenses within learning organizations. 
The adage “content is king” is popular for a reason. Without well-crafted and supported content 
any learning enterprise will fail. To continue with the analogy, context is the kingdom. A king is 
useless without a kingdom. The course should consist of well-structured content, but also must 
have learning supports around it to create a robust course. How to build this support can vary 
based on resources, culture, need, and implementation time. However, the recommendations 
below provide a structure of required elements that can be implemented in some form regardless 
of the individual composition of the learning organization. 


COURSE-LEVEL RECOMMENDATIONS 


DATA-SUPPORTED COURSES 


Technology should collect and support data within courses. To modernize courses and enable 
information sharing, learning technologies must be able to collect and output learning data. 
Several data standards are already in use. For example, xAPI is a popular method for capturing, 
standardizing, and sharing human performance data. 


LMS Data 
@9eee7ee ee 


Learning 2 
Technology Data Data ® mae 
eeeceo . @ecoeoeo anu 
Repository @ Maula 
® 


Performance Data 


EXAMPLE 


Use a standardized approach when capturing the interaction of students in the system (e.g., 
xAPI). An adapter can be developed to generate xAPI statements from existing or proprietary 
systems. Utilize existing xAPI Profiles Specification to enable successful and semantically 
interoperable xAPI implementation. Visualizations that use user data must be transparent to the 
users (i.e., users can understand how the data led to the visualization). When designing a 
visualization, relevant information to its intended user must be considered. The visualization 
should incorporate interactivity to allow for visual exploration and to prevent overwhelming the 
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users. Information Seeking Mantra can serve as a guide when designing interactive 
visualizations: overview first, zoom and filter, then details-on-demand. 


SUPPORT SELF-REGULATION 


Classes must support a learner’s self-regulation. Self-regulation is a broad term that refers to 
one’s ability to monitor and regulate their learning experiences. Self-regulation skills are 
essential for success in distributed learning environments. To successfully implement self- 


regulation, the follow steps must be supported. 
Phases of Self Regulated Learning 


e Phase 1: Learners should be allowed to plan and 
analyzing tasks for the class. 


e Phase 2: Learners must be able to perform tasks and 
enact their chosen strategies. @ 


e Phase 3: Learners must be able to monitor their 
performance and learning. 


i feyalicelalay=4 


e 


e Phase 4: Learners must be able to adapt future & 
learning efforts based on their observed results. 


Adapting 
It should be noted that these steps are not always easy in 
online settings. They can be facilitated by learning supports 
such as visual dashboards. During this process students often need to be monitored by faculty 
and to receive feedback, especially in Phase 2 when they are performing tasks. 


EXAMPLES 


A variety of ways to promote self-regulation exist in distributed learning contexts. One example 
is providing learners with goal-setting templates to support learners' free choice in their learning. 
It is a powerful method to help students unfold and frame their intrinsic goals or needs of 
learning (e.g., personal growth, meaningful relationships with others, or contributing to the 
community) instead of motivating them by extrinsic goals (e.g., fame, financial success, and 
physical appearance) or outside pressures (e.g., social comparison with peers). 


Examples of self-regulation support include: 


e Student-facing learning analytic dashboards, which can be effective tools for developing 
self-regulated learners because of their autonomous and informative feature (e.g., an 
adaptive recommendation system and interactive data visualization, etc.). 

e A recommendation system that aims for a more coarse-grained unit of learning (e.g., 
course modules) while allowing learners to explore additional materials. 

e An intelligent tutoring system that aims for the fine-grained solutions to the learning 
problems by providing specific feedback to learners as they work through the material. 
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SUPPORT MOTIVATION 


Students exhibit varied levels of motivation while learning content in distributed as well as face- 
to-face or blended learning environments. There are many theories of academic motivation that 
examine motivation from different perspectives. Some of these theories suggest the need to 
consider both internal and external motivational factors, such as an internal desire to succeed 
versus the need for an external motivator. Knowing that students approach distributed learning 
with varied levels and types of motivation can improve course designs by considering different 
strategies for supporting students’ motivation to learn the content. 


EXAMPLES 


Foster student motivation can take on many forms. However, supporting motivation may differ 
due to various types of content, context, and learner populations. For example, an individual 
taking a course that is critical to their career may feel an internal desire to succeed, whereas one 
taking a mandated course that has little relevance to their career may need more external 
motivators to succeed. 


One example of providing motivation is through the types of feedback and communication 
provided to learners. For example, instilling a growth mindset, which is the perspective that one 
can always improve their intelligence, can be beneficial. Praising effort and progress can provide 
benefits to learners. 


Personalized learning analytics applications such as student-facing learning analytics dashboards 
(SFLADs) can be helpful to increase learners’ motivation by affording them ownership over 
their own learning. These SFLADs can provide information such as their own learning 
progression, trajectory, and goals so they can make decisions on whether they want to continue a 
class or how they want to continue it. In short, SFLADs should empower students to become 
data-informed learners. Specifically, a well-design SFLAD should focus on designing the 
learning experience for learners which underscores the learners’ affective and non-cognitive 
skills (e.g., interpersonal skills, persistence, communication skills, and other "soft" skills). In 
addition, a variety of user experience design techniques (e.g., task analysis, questionnaires, 
interviews, usability testing, etc.) should be utilized to examine and evaluate the effectiveness of 
the SFLADs. 


SUPPORT INTERACTION THROUGH COURSE FEATURES 


Courses should be designed to support interactions. Building interactions with learning content is 
a staple task for many instructional designers. The goal of instruction is often for students to 
understand what the content means in the context of their own lives or careers, and therefore 
having students purposefully interact with said content can be effective for facilitating learning. 
This notion extends to distributed learning environments as well as face-to-face and blended 
learning environments. Student interaction with course content can come in many forms, 
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including augmented or virtual reality experiences or simulations (AR and VR, respectively), 
social media use, or cooperative learning opportunities. The appropriate use of each of these 
technologies or techniques will depend on the content and course being taught. 


There are also additional considerations regarding the use of social media and cooperative 
learning activities, which include, but are not limited to: 


e Making students aware of the risks and benefits of social media use in informal learning 
opportunities. 

e Training students in digital citizenship. 

e Protecting students’ anonymity, privacy, and choice when using social media sites for 
learning (e.g., FERPA considerations). 

e Accommodating assignments for students who are unwilling to engage in social media 
for learning. 

e Instructing students on the location and platform for each assignment (LMS or social 
media sites) so that there is no division of the location of learning materials. 


EXAMPLE 


Interactions with course content should be intentional and contextually relevant. Consider for 
example, AR or VR learning environments. AR or VR experiences or simulations have been 
used in a variety of fields. However, they have the potential to be time and cost intensive to 
develop, and therefore may not be appropriate for every use case. There is also a broad range of 
what these experiences or simulations may look like. For instance, a simple AR simulation may 
entail additional information showing on a screen when the camera picks up an object within its 
view, whereas a complex VR simulation could be used for training pilots. 


FOSTER A COMMUNITY OF INQUIRY 


Build interactions into the course. These interactions should include interacting with the content, 
but also with the instructor and other students. In the past, distributed courses were critiqued for 
a lack of interaction that made learners feel disconnected from the topic and class. Building 
interactions with course content helps this by building a community of inquiry within the course. 
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A community of inquiry is a connected group of learners with common goals. It fosters critical 
thinking, reflection, and discourse. To create this collaborative community, a class must have a 
feeling of presence (being present) associated with it. There are thee type: cognitive, teaching, 
and social. 


e Cognitive presence is the ability of the learner to manipulate the content in relation to 
their own lives. 

e Teaching presence refers to how well the instructor designs and facilitates the course. The 
instructor should be part of the course both visually in the form of video and as an 
interactive partner by providing feedback and responses to student questions. 

e Social presence refers to how cohesive the class of students is, or how well they openly 
communicate. This can be facilitated by providing communication outlets for students 
such as forums, discussions, or interactions on social systems such as Slack. However, 
these interactions should be monitored by the instructor to identify incorrect information 
or abuse. 


EXAMPLES 


Examples for increasing presence include: 


e Establishing a strong teaching presence as a moderator of the social and cognitive 
presences through using interaction opportunities and through establishing class 
administration early in the course cycle. 

e Helping students engage and collaborate through thoughtful, relevant, authentic group 
learning activities. 

e Segmenting content into sizes that suit the level of the learner (novice versus expert). 

e Providing strong evaluation rubrics and feedback that reward students for effort in 
multiple dimensions of class projects. 

e Helping students connect with peers through intentional community engagement 
activities. 

e Helping students transition from traditional lecture-based learning to collaborative 
learning through metacognitive strategies such as reflection. 

e Ensuring students’ peer-peer and peer-instructor interactions are safe and respectful. 
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PROVIDE SCAFFOLDING, INCLUDING MICROLEARNING AND FEEDBACK 


Provide guidance (scaffolding) within eLearning courses. There are many types of scaffolding, 
guidance, and feedback that can be critical elements of effective instruction in these contexts. 
Microlearning techniques can help in this process. Within microlearning, larger content is 
subdivided into smaller coherent topics. These topics can be provided to learners either in an 
ordered sequence or to give extra guidance if learners are struggling with a specific concept. 


EXAMPLES 


Examples of providing scaffolding and guidance include, but are not limited to: 


e Using authentic scenarios or cases that emphasize the general rules and principles of a 
discipline and highlighting how these rules and principles can be transferred into new 
situations. 

e Providing worked examples to lower extraneous cognitive load for novice learners. 

e Devising mechanisms for calling attention to unfamiliar vocabulary and principles or 
systems, such as flashcards, low-stakes quizzing, and demonstrations. 

e Setting realistic expectations for PBL/IL by remembering these tools promote meaning 
and understanding and are not a means for knowledge acquisition. 

e Providing human support and materials support, such as task overviews, scope of 
learning overviews (learning objectives or goal statements), task requirements, and ideal 
task representations so that students can emulate the ideal solution. 

e Providing microlearning opportunities to provide just-in-time teaching and support. 


Examples of providing feedback include, but are not limited to: 


e Delivering supportive feedback through prompts that provide opportunities for instructors 
to address domain-specific knowledge, guidance, and learner-strategies through 
explanations, references, visual aids, and project templates. 

e Encouraging reflective feedback using process maps, models of expert solutions, and 
hints and questions to guide learners through the process and to identify knowledge gaps. 

e Providing guidance to students by modeling good performance, providing task 
explanations, setting accomplishable task expectations, and providing expert guidance. 

e Delivering “feed-back” through giving specific, individual, timely, regular, detailed, 
purposeful, task-appropriate, assignment-directed, and future-driven statements that are 
expressed in concrete and understandable ways for all student assignments. 

e Using peer feedback for the benefits of providing students with other perspectives on 
their assignments. 
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DESIGNED ACCORDING TO HUMAN COGNITIVE PROCESSES 


The limitations of human cognitive processing should guide the design of instruction and 
instructional content. Research has shown that the working memory capacity is limited, 
instructional sequences should be designed to prevent overloading the working memory capacity. 
In addition to instructional sequencing, one must also consider the visual design of the materials 
presented. 


General recommendations from the research principles include: 


e Avoiding presenting complex text and visual images unless there is adequate time to 
process the information and guidance to facilitate understanding. 

e Avoiding the use of images unless they are needed and relevant to the explicit point being 
made. Images need to help the learner visualize the information. Otherwise, the images 
will hinder learning. 

e Presenting an exact narration of printed text should be avoided unless it is needed for a 
specific learner population (For example visually impaired learners). 

e Integrating narrative and visual information when presented at the same time. 

e Presenting a complex diagram or complex written document by highlighting key points 
when they are relevant. This might require taking a longer text or narrative information 
and turning it into a series of subtopics with relevant linking information highlighted in 
each. 


EXAMPLE 
A considerable number of instructional design principles can be used to help prevent overloading 
the working memory capacity and improve learning. Examples include the modality principle, 


spatial contiguity principle, temporal contiguity principle, worked example effect, signaling 
principle, and more. 


PROVIDE VIDEO CONTENT THAT INTEGRATES BEST PRACTICES 


Courses should use instructional videos that follow specific research-based best practice 
strategies. Video should not be just a long recording of a lecture as a replacement for a face-to- 
face lecture. Ideally these videos should be short (five to 10 minutes at most) and should model 
best practices, provide procedural knowledge (how-to knowledge), or brief, concise explanations 
of concepts. When producing these videos, follow best practices for production, including 
cognitive-based organizational principles such as using multiple modalities or signaling (visually 
highlighting) important concepts. Create notes of the videos, follow the notes, and provide a 
version of the notes to learners. 
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Video environment and instructor persona are also important. Stay professional but show your 
personality. Similarly, make sure that the environment around the instructor is professional yet 


relatable. 


EXAMPLE 


Examples of how to design effective 
instructional videos include, but are not 
limited to: 


e Ensuring that videos follow 
research-based instructional 
design principles based on 
human cognitive processes. 

e Establishing that video content is 
relevant to the topic and will 
increase student engagement and 
active learning. 

e Utilizing video messages with 
signaling and segmenting for 
learners to identify the important 
material quickly. 


e Using videos that adhere to ADA accessibility guidelines. 


Social 
(Be real) 


e Short 


¢ Model 
behaviors 


e Examples 


e Highlight key 
information 
e Provide notes 


¢ Personality over 
professionality 

¢ Relatable 
context 


e Considering the use of questions and reflection in videos for students to practice retrieval. 
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END OF REPORT APPENDIXES 


APPENDIX A 


Heuristic Evaluation Form — Organization Level 
Instructions 


This form is designed to help an organization determine how well it is 
meeting the needs of its distributed learning constituents at the institutional level. 
The person(s) completing this form should possess relevant expertise, such as a 
learning scientist or learning engineering. This form is not intended to be all- 
inclusive, but rather is designed to highlight the high-level requirements of an 
effective distributed learning organization. 


The organization can be categorized as one of four different ratings for each 
criterion, which are defined as follows: 


Rating Description 


Exemplary The organization provides ample resources compared 
to the current and projected needs. There are limited 
areas for improvement. 


Acceptable The organization provides the resources needed to 
meet current needs. However, there are still areas of 
potential improvement. 


Adequate The organization provides the minimum resources to 
meet current needs. More resources should be devoted 
to the criterion. This rating is considered the minimum 
needed for operation and may not be appropriate for 
sustained operations. 


Lacking The organization does not meet its constituents’ needs 
and is not serving its constituents well. 


20 


Organizations should strive to reach Acceptable or, if possible, Exemplary for all criteria. An 
organization with these ratings is likely in a good position to host effective distributed learning 
courses. Note that meeting these criteria alone does not indicate an effective distributed learning 
organization; rather, these criteria are indicative of having the capacity to be an effective 
distributed learning organization. 


An organization that currently provides distributed learning may find that they have criterion 
rated at Adequate. As noted, this indicates a minimum level of proficiency and the organization 
should conduct a needs analysis to find out the most appropriate way to improve in this criterion. 


Any criterion that is rated as Lacking should perform a needs analysis for that criterion to better 
understand why it is deficient, and then address these deficiencies before adding additional stress 
onto the organization. 
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Organization-Level Criterion Summary 


Topic 


Network of Education 
Expertise 


Established human-centered 
evaluation process 

Support institutional trust 
Support institution-student 


trust 


Student academic/social 
support services 


General support infrastructure 


General IT infrastructure 


Allow for flexible class sizes 


Explanation 


Is there an organized structure of individual with educational 
expertise throughout the organization. Are they structured so 
they can support each other? Are they structured to support 
strong educational practices throughout the organization? 


Does the institution have processes in place for formal or 
informal usability evaluations of courses or learning 
materials? 


Are there efforts, policies, and procedures in place to support 
and understand trust within the organization? 


Are there efforts, policies, and procedures in place to support 
institution-student trust? (e.g., building logical class 
structures, frequent communication, involving students in 
decision-making, defining clear policies, etc.) 


Are there ample student support services in place (e.g., 
providing students with administrative contact persons for 
individualized counsel, instituting institution-wide 
communication standards and evaluating communication 
flow at regular intervals, making students aware of financial 
obligations and sources of help, opportunities for inclusion in 
services offered by institutional services for persons with 
disabilities, etc.)? 


Are clear policies and supports in place so instructors know 
who to approach or what to do if they encounter issues with 
their courses? 


Is the general IT infrastructure available to support online 
courses at the scale the organization operates? For example, 
is there appropriate capacity to support the number of 
students accessing the system at once? Is an adequate 
analytics infrastructure in place? 


Class size should be guided by the nature of the content and 
the technology infrastructure available to support the course. 


pH) 


Learning Organization Heuristic Evaluation Form 


Use the form below to review the policies and procedures for the learning organization. Rate 
each category from 1 (Lacking) to 4 (Exemplary) based on the observed current state of the 
organization. In the following comment box, provide explanation of your rating and feedback for 
improvement. 


Category Rating (1-4) | Comments 


Network of 
Education 
Expertise 
Established 
human-centered 


evaluation 
process 


Support 
administrator- 
instructor trust 


Support 
institution- 
student trust 


Student 
academic/social 
support services 


General support 
infrastructure 


General IT 
infrastructure 


Allow for 
flexible class 
Sizes 
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APPENDIX B 


Heuristic Evaluation Form — Course Level 
Instructions 


This form is designed to inform on the potential for a course to meet learner needs. The person(s) 
completing this form should possess relevant expertise in learning science and learning 
engineering principles. This form is not intended to be all inclusive, but rather is designed to 
highlight the high-level requirements of an effective distributed learning course. In addition to 
the criteria listed here, a course should also adhere to standard best practices of accessibility and 
course design practices on assessment such as a clear link between content and assessment and 
clear explanations of assessment criteria. This form is not intended to be used as an instructor 
evaluation tool. 


The course can be categorized as one of four different ratings for each criterion, which are 
defined as follows: 


Rating Description 


Exemplary The course provides ample support for learners. There 
are limited areas for improvement. 


Acceptable The course provides the support needed to meet the 
needs of learners. However, there are still areas of 
potential improvement. 


Adequate The course provides the minimum support to meet 
learners needs. More support should be devoted to the 
criterion. This rating is considered the minimum 
needed for course delivery and may not be at 
appropriate levels for all learners. 


Lacking The course does not meet learners’ needs on this 
criterion in the current state. 
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Ideally, courses will have ratings of Acceptable or Exemplary for all criterion. A course with 
these ratings is likely in a good position to facilitate effective distributed learning. Note that 
meeting these criteria alone does not indicate an effective distributed learning course. These 
criteria are indicative of having the capacity to be an effective distributed learning course. 
Effective course facilitation is essential in the learning process and largely unaccounted for by 
this evaluation form. 


A course may have criterion rated Adequate. This indicates a minimum level of proficiency and 
the course designer should conduct a needs analysis to find out the most appropriate way to 
improve in this criterion. 


Any criterion that is rated as Lacking should perform a needs analysis of that criterion to better 
understand why it is deficient before the course is provided to learners. 


Course-Level Criterion Summary 


Course Structure and Alignment 
Topic Explanation 


Fundamental data collection for analytics Does the course allow for the 
collection of fundamental data that 
could be used for analytics? 
Examples (non-inclusive list) may 
include quiz scores, final grades, 
attendance, etc. 


Support self-regulation Does the course include design 
features that support student self- 
regulation? Examples may include 
features such as a calendar of 
assignments and due dates, or an 
analytics dashboard. 


Support motivation Does the course include design 
features to support student 
motivation? Potential examples 
include tying the content to their 
career field or personal interests. 


Support interaction with course features Does the course contain adequate 
features to support student 
interaction with the content (e.g., 
virtual reality, augmented reality, 
social media, simulations, etc.)? 
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Fosters a Community of Inquiry 


Content Design 


Provides scaffolding, including microlearning and 
feedback 


Designed according to human cognitive processes 


Provides video content integrates best practices 


Does the course support cognitive, 
social, and teaching presence? 


Does the course include appropriate 
scaffolding (e.g., extra readings, 
videos, just-in-time teaching, or 
microlearning opportunities)? 


Does the course include frequent 
opportunities for the student to 
receive formative feedback? 


There are a considerable number of 
instructional design principles that 
can be used to help prevent 
overloading the working memory 
capacity and improve learning (e.g., 
the modality principle, spatial 
contiguity principle, temporal 
contiguity principle, worked example 
effect, signaling principle, etc.). 


Videos should follow research-based 
instructional design principles based 
on human cognitive processes. In 
addition, videos can include content 
that will increase student 
engagement and active learning, 
adhere to ADA accessibility 
guidelines, and use questions and 
reflection for students to practice 
retrieval. 
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Course - Heuristic Learning Science Evaluation Form 


Use the form below to review the policies and procedures for the learning organization. Rate 
each category from | (Lacking) to 4 (Exemplary) based on the observed current state of the 
organization. In the following comment box, provide explanation of your rating and feedback for 
improvement. 


Category Rating (1-4) | Comments 


Course Design 


Fundamental 
data collection 
for analytics 


Support self- 
regulation 


Support 
motivation 


Support 
interaction with 
course features 


Fosters a 
Community of 


Inquiry 
Content Design 


Provides 
scaffolding, 
including 
microlearning 
and feedback 


Designed 
according to 
human 
cognitive 
processes 


Provides video 
content 
integrates best 


practices 


